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Background/Context: 


Nearly one in five Americans does not graduate from high school (U.S. Department of 
Education, 2017) and risk of dropout disproportionally affects economically disadvantaged, 
racial/ethnic minorities. Dropout occurs for many reasons but usually results from cumulative 
disengagement with school (Fine, 1991; Orfield et al., 2004). The middle school years offer a 
critical intervention point for two reasons: (1) simple criteria (e.g., course failure) during middle 
school predict dropout (Balfanz, Herzong, & Maclver, 2007; Neild, Balfanz, & Herzog, 2007); 
and (2) intervening prior to high school is ideal because disengagement rarely results from 
students’ diminished desire to do well academically but rather from a failure to connect long- 
term future goals to immediate performance and invest appropriate self-regulatory effort 
(Oyserman, 2012, 2015). 


Identity-Based Motivation (IBM) interventions that make the future feel close (by focusing 
students on their future selves) and school feel like the path to get there have been shown to 
improve academic outcomes (Oyserman, 2012, 2015). In a randomized control trial conducted in 
Detroit (Oyserman et al., 2006), one brief, IBM intervention administered in ge grade called 
School-to-Jobs demonstrated improvements in self-regulatory behaviors (decreased unexcused 
absences, increased time spent doing homework, increased initiative in classroom; decreased 
disruptive behaviors) and academic outcomes (core subject GPA, test scores, retention) among 
low-income, at-risk eighth-grade students. Significant effects persisted or grew larger across the 
transition to high school two years post-intervention and effects were mediated by changes in 
IBM. Although this study provided a rigorous test of theory, the intervention was led by external 
trainers and is not scalable. 


Objective/Research Questions: 


Under a development grant from the Institute of Education Sciences (IES), and a research- 
practice partnership with Chicago Public Schools (CPS), we developed and pilot-tested a 
scalable teacher-led, teacher-trained adaptation of the intervention called Pathways to Success 
(Pathways). This paper highlights refinements to enhance usability/feasibility of implementation. 
A pilot study examined the following research questions: 

1. To what extent can eighth-grade teachers implement Pathways with fidelity? 

2. What is “impact” of Pathways on eighth-grade student academic outcomes? 


Setting: 

CPS is the third-largest U.S. school district (480 K—8 schools), and mirrors other U.S. urban 
districts: 74% graduation rate; a majority (80%) of students come from low-income families and 
are minorities (e.g., 38% Black, 47% Hispanic); 17% have limited English proficiency. 


Population/Participants/Subjects: 


Across two school years (2014-15, 2015-16) five K-8 elementary schools, 15 eighth-grade 
teachers, and 380 eighth-grade students participated (see Table B-1 for characteristics of 
participating schools). 


Intervention: 
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Pathways involves small-group activities presented twice weekly for 30-45 minutes in class over 
the first six weeks of the school year. Each session involves an activity focused on a particular 
take-home point. Activities build on one another and activate each of the elements of IBM. 
Implementation is guided by a manual for teachers, and demonstrated via a video library. The 
theory of change (Figure B-1) predicts that Pathways will increase IBM (possible identities and 
strategies to attain them and interpretations of difficulty). IBM, in turn, facilitates self-regulatory 
behaviors, resulting in better academic outcomes. 


Research Design: 


We iteratively refined and tested Pathways implementation resources. First, we simplified the 
manual to facilitate usability. Second, we developed sustainable materials to implement each 
activity, requiring little-to-no continued costs of implementation. Third, we operationalized what 
quality delivery looks like in each session. Fourth, we created sensitive, quality-of-delivery 
fidelity measures that can be reliably coded from video. Fifth, we created a website for teachers, 
including the implementation manual, all handouts, PowerPoint presentation, and videos for each 
session. Sixth, we produced a set of video clips that articulate the underlying theory and how it 
relates to specific activities. Seventh, we conducted the same iterative process for the trainer 
manual and materials. 


To address the research questions, fidelity was assessed via video-observation and student self- 
report. Student outcomes were assessed via a propensity-matched quasi-experimental design to 
minimize selection bias and maximize internal validity in the absence of random assignment. 


Data Collection: 


The study team video-recorded all 12 sessions for all teachers. Following Durlak and Dupre 
(2008) we assessed how much of the intended intensity and duration of intervention was 
delivered (dosage), whether delivery followed protocol (adherence), whether active ingredients 
were delivered (quality), whether students responded to ingredients as intended (student 
responsiveness), and whether students understood the intervention (fidelity of receipt). 


Student administrative data from prior years and during eighth-grade were collected from the 
district to assess outcomes. 


Analysis: 


First, we calculated propensity scores using a selection model. Second, we employed 1:1 
matching without replacement to match students who participated in treatment schools with 
similar students in non-treatment schools within the same network (sub-geographic region of 
Chicago). Third, we calculated the average treatment effect on the treated, using a regression to 
estimate effects of participating in Pathways, controlling for student characteristics, cohort, and 
network. 


Findings/Results: 


Teachers implemented with moderate-to-high fidelity, and implementation resulted in positive 
academic outcomes for students. Durlak and Dupree (2008) suggest that most interventions 
achieve fidelity of implementation if they implement with 60% or higher fidelity, and note that 
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few studies attain greater than 80%. Fidelity for this study averaged 79%, ranging from 69 to 
91%. Analyses of student outcomes are promising: Treatment and matched-comparisons students 
did not differ at baseline (d’s=0-0.09) but students in schools that participated in Pathways 
during the 2014—15 and 2015-16 school years had significantly higher GPAs (d=0.35), were less 
likely to fail a course (Cox Index=-0.35) and averaged fewer course failures (Cox Index=-0.30). 
We did not find impacts on attendance—the unexcused absence rate was low (2%) for both 
treatment and comparison groups, reflecting the 93% districtwide attendance rate (Chicago 
Public Schools, 2016). 


Conclusions: 


Though preliminary, our study suggests the following conclusions: 
= Eighth-grade teachers can implement Pathways with moderate-to-high fidelity. 
« Pilot analyses of outcomes show promise for students—improving course performance 
and reducing risk of course failure in eighth grade. 
= Future research employing an experimental design is needed to test the impact of 
Pathways on student outcomes. 
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Appendix B. Tables and Figures 


Table B-1. Characteristics of participating schools 


School Characteristic School 1 School 2 School 3 School 4 School 5 
Grades Pre K-8 K-8 Pre K-8 Pre K-8 K-8 
Total Number of Students 1,580 4D 286 817 806 
Number of Participating Eighth-Grade 
5 3 1 3 3 
Teachers 
Number of Participating Eighth-Grade Students 103 84 21 82 98 
Schoolwide Percentage Hispanic/Latino 96% 31% 0% 67% 28% 
Schoolwide Percentage Black 1% 22% 100% 31% 1% 
Schoolwide Percentage Asian 0% 38% 0% 0% 6% 
Schoolwide Percentage White 2% 7% 0% 1% 60% 
Schoolwide Percentage Other 1% 2% 0% 1% 5% 
Schoolwide Percentage Low Income 98% 94% 95% 99% 37% 
Schoolwide Percentage Limited English 43% 43% 0% 29% 21% 
Schoolwide Mobility Rate 21% 14% 37% 28% 11% 
Eighth-Grade NWEA Reading Growth 2014 99 98 5 om) 98 
Percentile 
Eighth-Grade NWEA Math Growth 2014 94 66 98 B 91 
Percentile 
Eighth-Grade Student Attainment Reading 79 88 1 59 92 
Percentile 
Eighth-Grade Student Attainment Math 59 8] 37 59 9] 
Percentile 
Northern eg Far North 
Geographic Region of City Northwest Side : Southeast West Side ; 
Side ; Side 
Side 
Number Suspensions / 100 Students 0.6 3.6 14.1 0.5 


NOTE: The national average for NWEA Growth in Reading and Math is the 50th percentile, and the national average student attainment in reading and math is 
the 50th percentile. 
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Figure B-1. Pathways Theory of Change 
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